A digital color image usually consists of an array of pixel values representing the intensity of the image at each point on a regular grid. Typically, three colors are used to generate the image. At each point on the grid the intensity of each of these colors is specified, thereby specifying both the intensity and color of the image at that grid point.
Conventional color photography records the relevant image data by utilizing three overlapping color sensing layers having sensitivities in different regions of the spectrum (usually red, green, and blue). Digital cameras, in contrast, typically utilize one array of sensors in a single “layer”.
When only one sensor array is used to detect color images, only one color may be detected at any given sensor location. As a result, these sensors do not produce a color image in the traditional sense, but rather a collection of individual color samples, which depend upon the assignment of color filters to individual sensors. This assignment is referred to as the color filter array (CFA) or the color mosaic pattern. To produce a true color image, with a full set of color samples (usually red, green and blue) at each sampling location, a substantial amount of computation is required to estimate the missing information, since only a single color was originally sensed at each location in the array. This operation is typically referred to as “demosaicing”.
To generate the missing information, information from neighboring pixels in the image sensor must be used. A number of algorithms have been put forward in an attempt to provide the missing information while minimizing artifacts resulting from the estimation process. The simplest algorithms interpolate the sensor data from like color sensors to provide the missing information. These algorithms treat the red sensors as being independent from the green sensors, and so on. To provide a red value at a given location, the values measured by the red sensors in the region of that location are interpolated. This approach requires that the image be low-pass filtered. Such filtering reduces the image resolution below the pixel resolution of the underlying sensor array. This lost resolution cannot be recovered.
To avoid this loss in resolution, less aggressive optical low-pass filtering is used in some higher-end cameras. However, in such systems, the color sensors may no longer be treated as independent. For example, Wober, et al. (U.S. Pat. No. 5,475,769) describe a method for generating the missing color information by computing a weighted average of the pixel values in the neighborhood of the pixel whose missing color information is being computed. This method weights values from all of the color sensors, not just the color being reconstructed. However, even this approach leaves much to be desired since it utilizes one set of weights for all images.
A single pixel array may be viewed as consisting of a number of separate planes of pixels in which each plane has sensors for the same color. Since the pixels do not overlay, the sensors in the various planes are at different locations. Systems that take weighted averages across more than one plane make use of the statistical dependencies between these sample locations. In effect, the blurring of an image by the camera optics allows an image edge that falls on one color plane precisely on the sensors of that plane to also be seen in the other color planes because the image is spread by blurring onto the sensors in the other color plane. Since the statistical dependencies between the various color planes depend on the amount of blur introduced by the camera optics, an optimal algorithm must take into account the physical camera settings. Accordingly, a single set of weight functions will not provide an optimal estimation of the missing information.
The statistical dependencies also depend on the source of illumination. Different illumination sources have different spectra. The pixel filters have broad pass-bands centered at the red, green, and blue wavelengths. In the absence of any image blurring, the response of any given pixel is determined by its color filter, the reflectivity of the corresponding point in the scene being photographed, and the light spectrum incident on that point from the illumination source. The blurring provided by the camera optics mixes the light between the pixels. Hence, the statistical dependencies, in general, depend both on the illumination source and the camera optics. Prior art methods for converting the pixel array data to a fully sampled color digital image do not take the illumination source into account.
Broadly, it is the object of the present invention to provide an improved image processing method for converting data from a pixel array having non-overlapping sensors to a fully sampled digital image.
It is a further object of the present invention to provide a conversion method that corrects for the camera's optical system.
It is a still further object of the present invention to provide a conversion method that corrects for the source of illumination.
These and other objects of the present invention will become apparent to those skilled in the art from the following detailed description of the invention and the accompanying drawings.